Pass multiprocessing parameters through the shell #952
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
There is a timeout test error of 30s for the additional test I added with the |
…hat it finishes within the timeout limit
…er as 0 so the user will need to turn multiprocessing explicilty on
AaronSpieler
approved these changes
Jul 27, 2020
kashif
pushed a commit
to kashif/gluon-ts
that referenced
this pull request
Oct 10, 2020
* Passing num_workers, num_shuffle_batches and is_cached through the shell * Pass num_workers, num_prefetch and shuffle_buffer_length through the shell * Fix black * Fix spacing * Revert back to shuffle_buffer_length * Decreasing number of training epochs in the smoke test for MQCNN so that it finishes within the timeout limit * Default to num_workers 2. * Setting default num_workers = None to be set in the parallelized_loader as 0 so the user will need to turn multiprocessing explicilty on Co-authored-by: Danielle Robinson <dmmaddix@amazon.com> Co-authored-by: Aaron Spieler <25365592+AaronSpieler@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
num_workers,num_prefetch,shuffle_buffer_lengthashyperparameterkeys through the shellrun_trainmethod to read these keys in from the hyper-parameters and set their valuesnum_workerswe set to be the minimum of inputed value if set and if not input default 4 and the sort of the number of cpusparallelized_loader.pyListDatasetandFileDataset, respectivelyTesting
MQCNNEstimatorasforecaster_typetotest_train_shellto extend the test for trainable algorithms unlike the currentMeanPredictorMQCNNEstimatorBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.